Collecting patient-reported outcome measures in the electronic health record: Lessons from the NIH pragmatic trials Collaboratory.

The NIH Pragmatic Trials Collaboratory supports the design and conduct of 27 embedded pragmatic clinical trials, and many of the studies collect patient reported outcome measures as primary or secondary outcomes. Study teams have encountered challenges in the collection of these measures, including challenges related to competing health care system priorities, clinician's buy-in for adoption of patient-reported outcome measures, low adoption and reach of technology in low resource settings, and lack of consensus and standardization of patient-reported outcome measure selection and administration in the electronic health record. In this article, we share case examples and lessons learned, and suggest that, when using patient-reported outcome measures for embedded pragmatic clinical trials, investigators must make important decisions about whether to use data collected from the participating health system's electronic health record, integrate externally collected patient-reported outcome data into the electronic health record, or collect these data in separate systems for their studies.

Monitoring and responding to signals of suicidal ideation in pragmatic clinical trials: Lessons from the GRACE trial for Chronic Sickle Cell Disease Pain.

Sickle cell disease (SCD) is a hemoglobin disorder and the most common genetic disorder that affects 100,000 Americans and millions worldwide. Adults living with SCD have pain so severe that it often requires opioids to keep it in control. Depression is a major global public health concern associated with an increased risk in chronic medical disorders, including in adults living with sickle cell disease (SCD). A strong relationship exists between suicidal ideation, suicide attempts, and depression. Researchers enrolling adults living with SCD in pragmatic clinical trials are obligated to design their methods to deliberately monitor and respond to symptoms related to depression and suicidal ideation. This will offer increased protection for their participants and help clinical investigators meet their fiduciary duties. This article presents a review of this sociotechnical milieu that highlights, analyzes, and offers recommendations to address ethical considerations in the development of protocols, procedures, and monitoring activities related to suicidality in depressed patients in a pragmatic clinical trial.

Potential bias and lack of generalizability in electronic health record data: reflections on health equity from the National Institutes of Health Pragmatic Trials Collaboratory.

Embedded pragmatic clinical trials (ePCTs) play a vital role in addressing current population health problems, and their use of electronic health record (EHR) systems promises efficiencies that will increase the speed and volume of relevant and generalizable research. However, as the number of ePCTs using EHR-derived data grows, so does the risk that research will become more vulnerable to biases due to differences in data capture and access to care for different subsets of the population, thereby propagating inequities in health and the healthcare system. We identify 3 challenges-incomplete and variable capture of data on social determinants of health, lack of representation of vulnerable populations that do not access or receive treatment, and data loss due to variable use of technology-that exacerbate bias when working with EHR data and offer recommendations and examples of ways to actively mitigate bias.

Equity and bias in electronic health records data.

Embedded pragmatic clinical trials (ePCTs) are conducted during routine clinical care and have the potential to increase knowledge about the effectiveness of interventions under real world conditions. However, many pragmatic trials rely on data from the electronic health record (EHR) data, which are subject to bias from incomplete data, poor data quality, lack of representation from people who are medically underserved, and implicit bias in EHR design. This commentary examines how the use of EHR data might exacerbate bias and potentially increase health inequities. We offer recommendations for how to increase generalizability of ePCT results and begin to mitigate bias to promote health equity.

Intervention delivery for embedded pragmatic clinical trials: Development of a tool to measure complexity.

BACKGROUND: Conducting an embedded pragmatic clinical trial in the workflow of a healthcare system is a complex endeavor. The complexity of the intervention delivery can have implications for study planning, ability to maintain fidelity to the intervention during the trial, and/or ability to detect meaningful differences in outcomes. METHODS: We conducted a literature review, developed a tool, and conducted two rounds of phone calls with NIH Pragmatic Trials Collaboratory Demonstration Project principal investigators to develop the Intervention Delivery Complexity Tool. After refining the tool, we piloted it with Collaboratory demonstration projects and developed an online version of the tool using the R Shiny application (https://duke-som.shinyapps.io/ICT-ePCT/). RESULTS: The 6-item tool consists of internal and external factors. Internal factors pertain to the intervention itself and include workflow, training, and the number of intervention components. External factors are related to intervention delivery at the system level including differences in healthcare systems, the dependency on setting for implementation, and the number of steps between the intervention and the outcome. CONCLUSION: The Intervention Delivery Complexity Tool was developed as a standard way to overcome communication challenges of intervention delivery within an embedded pragmatic trial. This version of the tool is most likely to be useful to the trial team and its health system partners during trial planning and conduct. We expect further evolution of the tool as more pragmatic trials are conducted and feedback is received on its performance outside of the NIH Pragmatic Trials Collaboratory.

Patients' Experiences With Staphylococcus aureus and Gram-Negative Bacterial Bloodstream Infections: Results From Cognitive Interviews to Inform Assessment of Health-Related Quality of Life.

BACKGROUND: We previously conducted a concept elicitation study on the impact of Staphylococcus aureus and gram-negative bacterial bloodstream infections (SAB/GNB) on health-related quality of life (HRQoL) from the patient's perspective and found significant impacts on HRQoL, particularly in the physical and functional domains. Using this information and following guidance on the development of patient-reported outcome (PRO) measures, we determined which combination of measures and items (ie, specific questions) would be most appropriate in a survey assessing HRQoL in bloodstream infections. METHODS: We selected a variety of measures/items from the Patient-Reported Outcomes Measurement Information System (PROMIS) representing different domains. We purposefully sampled patients ~6-12 weeks post-SAB/GNB and conducted 2 rounds of cognitive interviews to refine the survey by exploring patients' understanding of items and answer selection as well as relevance for capturing HRQoL. RESULTS: We interviewed 17 SAB/GNB patients. Based on the first round of cognitive interviews (n = 10), we revised the survey. After round 2 of cognitive interviewing (n = 7), we finalized the survey to include 10 different PROMIS short forms/measures of the most salient HRQoL domains and 2 adapted questions (41 items total) that were found to adequately capture HRQoL. CONCLUSIONS: We developed a survey from well-established PRO measures that captures what matters most to SAB/GNB patients as they recover. This survey, uniquely tailored to bloodstream infections, can be used to assess these meaningful, important HRQoL outcomes in clinical trials and in patient care. Engaging patients is crucial to developing treatments for bloodstream infections.

Responding to signals of mental and behavioral health risk in pragmatic clinical trials: Ethical obligations in a healthcare ecosystem.

BACKGROUND: Ethical responsibilities for monitoring and responding to signals of behavioral and mental health risk (such as suicidal ideation, opioid use disorder, or depression) in general clinical research have been described; however, pragmatic clinical trials (PCTs) raise new contextual challenges. METHODS: We use our experience with the PRISM (Pragmatic and Implementation Studies for the Management of Pain to Reduce Opioid Prescribing) program, which is a component of the Helping End Addiction Long-Term (HEAL) Initiative, to provide examples of research studying nonpharmacologic interventions for pain that collect sensitive data. Members of the PRISM Ethics and Regulatory Core and Patient-Centered Outcome Core Working Group discussed and refined considerations and recommendations. RESULTS: PCT researchers can help identify the extent of their ethical obligations to monitor and respond to signals of potential behavioral and mental health risks by understanding and aligning stakeholder expectations; considering characteristics of the trial and study population; defining triggers, thresholds, and responsibilities for action; identifying appropriate response mechanisms and capabilities; integrating responses with health systems; and addressing privacy. Based on such an assessment, researchers should proactively identify if, when, and how a response will be triggered. Doing so necessitates that stakeholders understand their roles in managing such risks. Finally, consent forms and other study disclosures should clearly state what if any responses might be taken. CONCLUSION: Early and ongoing bi-directional communication with relevant stakeholders is critical to identifying and meeting the ethical challenges for PCTs when managing and responding to behavioral and mental health data that potentially signal elevated risk to individuals.

Enhancing the use of EHR systems for pragmatic embedded research: lessons from the NIH Health Care Systems Research Collaboratory.

OBJECTIVE: We identified challenges and solutions to using electronic health record (EHR) systems for the design and conduct of pragmatic research. MATERIALS AND METHODS: Since 2012, the Health Care Systems Research Collaboratory has served as the resource coordinating center for 21 pragmatic clinical trial demonstration projects. The EHR Core working group invited these demonstration projects to complete a written semistructured survey and used an inductive approach to review responses and identify EHR-related challenges and suggested EHR enhancements. RESULTS: We received survey responses from 20 projects and identified 21 challenges that fell into 6 broad themes: (1) inadequate collection of patient-reported outcome data, (2) lack of structured data collection, (3) data standardization, (4) resources to support customization of EHRs, (5) difficulties aggregating data across sites, and (6) accessing EHR data. DISCUSSION: Based on these findings, we formulated 6 prerequisites for PCTs that would enable the conduct of pragmatic research: (1) integrate the collection of patient-centered data into EHR systems, (2) facilitate structured research data collection by leveraging standard EHR functions, usable interfaces, and standard workflows, (3) support the creation of high-quality research data by using standards, (4) ensure adequate IT staff to support embedded research, (5) create aggregate, multidata type resources for multisite trials, and (6) create re-usable and automated queries. CONCLUSION: We are hopeful our collection of specific EHR challenges and research needs will drive health system leaders, policymakers, and EHR designers to support these suggestions to improve our national capacity for generating real-world evidence.

Accounting for quality improvement during the conduct of embedded pragmatic clinical trials within healthcare systems: NIH Collaboratory case studies.

Embedded pragmatic clinical trials (ePCTs) and quality improvement (QI) activities often occur simultaneously within healthcare systems (HCSs). Embedded PCTs within HCSs are conducted to test interventions and provide evidence that may impact public health, health system operations, and quality of care. They are larger and more broadly generalizable than QI initiatives, and may generate what is considered high-quality evidence for potential use in care and clinical practice guidelines. QI initiatives often co-occur with ePCTs and address the same high-impact health questions, and this co-occurrence may dilute or confound the ability to detect change as a result of the ePCT intervention. During the design, pilot, and conduct phases of the large-scale NIH Collaboratory Demonstration ePCTs, many QI initiatives occurred at the same time within the HCSs. Although the challenges varied across the projects, some common, generalizable strategies and solutions emerged, and we share these as case studies. KEY LESSONS: Study teams often need to monitor, adapt, and respond to QI during design and the course of the trial. Routine collaboration between ePCT researchers and health systems stakeholders throughout the trial can help ensure research and QI are optimally aligned to support high-quality patient-centered care.

Addressing guideline and policy changes during pragmatic clinical trials.

While conducting a set of large-scale multi-site pragmatic clinical trials involving high-impact public health issues such as end-stage renal disease, opioid use, and colorectal cancer, there were substantial changes to both policies and guidelines relevant to the trials. These external changes gave rise to unexpected challenges for the trials, including decisions regarding how to respond to new clinical practice guidelines, increased difficulty in implementing trial interventions, achieving separation between treatment groups, and differential responses across sites. In this article, we describe these challenges and the approaches used to address them. When deliberating appropriate action in the face of external changes during a pragmatic clinical trial, we recommend considering the well-being of the participants, clinical equipoise, and the strength and quality of the evidence associated with the change; involving those charged with data and safety monitoring; and where possible, planning for potential external changes as the trial is being designed. Any solution must balance the primary obligation to protect the well-being of participants with the secondary obligation to protect the integrity of the trial in order to gain meaningful answers to important public health questions.

Pragmatic clinical trials embedded in healthcare systems: generalizable lessons from the NIH Collaboratory.

BACKGROUND: The clinical research enterprise is not producing the evidence decision makers arguably need in a timely and cost effective manner; research currently involves the use of labor-intensive parallel systems that are separate from clinical care. The emergence of pragmatic clinical trials (PCTs) poses a possible solution: these large-scale trials are embedded within routine clinical care and often involve cluster randomization of hospitals, clinics, primary care providers, etc. Interventions can be implemented by health system personnel through usual communication channels and quality improvement infrastructure, and data collected as part of routine clinical care. However, experience with these trials is nascent and best practices regarding design operational, analytic, and reporting methodologies are undeveloped. METHODS: To strengthen the national capacity to implement cost-effective, large-scale PCTs, the Common Fund of the National Institutes of Health created the Health Care Systems Research Collaboratory (Collaboratory) to support the design, execution, and dissemination of a series of demonstration projects using a pragmatic research design. RESULTS: In this article, we will describe the Collaboratory, highlight some of the challenges encountered and solutions developed thus far, and discuss remaining barriers and opportunities for large-scale evidence generation using PCTs. CONCLUSION: A planning phase is critical, and even with careful planning, new challenges arise during execution; comparisons between arms can be complicated by unanticipated changes. Early and ongoing engagement with both health care system leaders and front-line clinicians is critical for success. There is also marked uncertainty when applying existing ethical and regulatory frameworks to PCTS, and using existing electronic health records for data capture adds complexity.

Case Studies from the Clinic: Initiating and Implementing Patient-Reported Outcome Measures.

INTRODUCTION: Self-reporting by patients though the use of electronic patient-reported outcome (PRO) measures has been shown to use increase patient satisfaction with care, and improve patient-provider communication, symptom management, and health quality. Additionally, PROs are increasingly used in research to expand understanding regarding the relative risks, benefits, and burdens of interventions. While experience embedding patient-reported outcomes (PROs) into registries and clinical workflow is growing, there is little in the literature to guide those interested in incorporating PROs into routine clinical care and for use in research. CASE DESCRIPTIONS: The NIH Health Care Systems Research Collaboratory PRO Core interviewed investigators from seven programs to get their first-hand experiences on the incorporation of PROs for both care and research, and the investigators have contributed to this manuscript as authors. FINDINGS: We use these case studies to present practical approaches to initiating and implementing PROS, including instrument selection, tips for integrating PRO collection systems into clinical workflow, considerations for user experience and data collection, and the methods to assess and monitor quality. CONCLUSION: Because the decision to initiate and implement PRO collection impacts many different stakeholders, the solution requires collaboration among the involved parties, careful planning, and integration into clinical workflow.

Privacy and confidentiality in pragmatic clinical trials.

With pragmatic clinical trials, an opportunity exists to answer important questions about the relative risks, burdens, and benefits of therapeutic interventions. However, concerns about protecting the privacy of this information are significant and must be balanced with the imperative to learn from the data gathered in routine clinical practice. Traditional privacy protections for research uses of identifiable information rely disproportionately on informed consent or authorizations, based on a presumption that this is necessary to fulfill ethical principles of respect for persons. But frequently, the ideal of informed consent is not realized in its implementation. Moreover, the principle of respect for persons­which encompasses their interests in health information privacy­can be honored through other mechanisms. Data anonymization also plays a role in protecting privacy but is not suitable for all research, particularly pragmatic clinical trials. In this article, we explore both the ethical foundation and regulatory framework intended to protect privacy in pragmatic clinical trials. We then review examples of novel approaches to respecting persons in research that may have the added benefit of honoring patient privacy considerations.

Oversight on the borderline: Quality improvement and pragmatic research.

Pragmatic research that compares interventions to improve the organization and delivery of health care may overlap, in both goals and methods, with quality improvement activities. When activities have attributes of both research and quality improvement, confusion often arises about what ethical oversight is, or should be, required. For routine quality improvement, in which the delivery of health care is modified in minor ways that create only minimal risks, oversight by local clinical or administrative leaders utilizing institutional policies may be sufficient. However, additional consideration should be given to activities that go beyond routine, local quality improvement to first determine whether such non-routine activities constitute research or quality improvement and, in either case, to ensure that independent oversight will occur. This should promote rigor, transparency, and protection of patients' and clinicians' rights, well-being, and privacy in all such activities. Specifically, we recommend that (1) health care organizations should have systematic policies and processes for designating activities as routine quality improvement, non-routine quality improvement, or quality improvement research and determining what oversight each will receive. (2) Health care organizations should have formal and explicit oversight processes for non-routine quality improvement activities that may include input from institutional quality improvement experts, health services researchers, administrators, clinicians, patient representatives, and those experienced in the ethics review of health care activities. (3) Quality improvement research requires review by an institutional review board; for such review to be effective, institutional review boards should develop particular expertise in assessing quality improvement research. (4) Stakeholders should be included in the review of non-routine quality improvement and quality improvement-related research proposals. Only by doing so will we optimally leverage both pragmatic research on health care delivery and local implementation through quality improvement as complementary activities for improving health.

Ethical responsibilities toward indirect and collateral participants in pragmatic clinical trials.

Pragmatic clinical trials are designed to inform decision makers about the benefits, burdens, and risks of health interventions in real-world settings. Pragmatic clinical trials often use for research purposes data collected in the course of clinical practice. The distinctive features of pragmatic clinical trials demand fresh thinking about what is required to act properly toward people affected by their conduct, in ways that go beyond ensuring the protection of rights and welfare for "human research subjects" under conventional research ethics regulations. To stimulate such work, we propose to distinguish among categories of research participants in pragmatic clinical trials as follows: Direct participants: (1) individuals being directly intervened upon and/or (2) individuals from whom personal identifiable data are being collected for the purposes of the pragmatic clinical trial. Indirect participants: individuals who are (1) not identified as direct participants and (2) whose rights and welfare may be affected by the intervention through their routine exposure to the environment in which the intervention is being deployed. Collateral participants: patient groups and other stakeholder communities who may be otherwise affected by the occurrence and findings of the pragmatic clinical trial. We illustrate these distinctions with case examples and discuss the distinctive responsibilities of researchers and pragmatic clinical trial leadership toward each type of participant. We suggest that pragmatic clinical trial investigators, institutional review boards, health systems leaders, and others engaged in the research enterprise work together to identify these participants. For indirect participants, risks and benefits to which they are exposed should be weighed to ensure that their rights and welfare are protected accordingly, and communication strategies should be considered to help them make well-informed decisions. Collateral participants could provide input on the design, planning, and conduct of a pragmatic clinical trial and offer insights regarding the best way to communicate the trial's results to their constituencies.

Improved patient care and reduced costs: results of Baptist Health Systems' acute coronary syndromes project.

Acute coronary syndromes are deadly conditions that affect millions of people across the nation. As our data show, significant opportunities for quality improvement exist. Heathcare professionals, especially those involved in cardiovascular management, should spearhead quality improvement initiatives at their facilities. These efforts may result in decreased costs, increased patient satisfaction, and more importantly, decreased mortality.

Contemporary utilization and outcomes of intra-aortic balloon counterpulsation in acute myocardial infarction: the benchmark registry.

OBJECTIVES: We sought to examine contemporary utilization patterns and clinical outcomes in patients with acute myocardial infarction (AMI) requiring intra-aortic balloon pump (IABP) counterpulsation. BACKGROUND: Despite increasing experience with and broadened indications for intra-aortic counterpulsation, the current indications, associated complications, and clinical outcomes of IABP use in AMI are unknown. METHODS: Between June 1996 and August 2001, data were prospectively collected from 22,663 consecutive patients treated with aortic counterpulsation at 250 medical centers worldwide; 5,495 of these patients had AMI. RESULTS: Placement of an IABP in AMI patients was most frequently indicated for cardiogenic shock (27.3%), hemodynamic support during catheterization and/or angioplasty (27.2%) or prior to high-risk surgery (11.2%), mechanical complications of AMI (11.7%), and refractory post-myocardial infarction unstable angina (10.0%). Balloon insertions were successful in 97.7% of patients. Diagnostic catheterization was performed in 96% of patients, and 83% underwent coronary revascularization before hospital discharge. The in-hospital mortality rate was 20.0% (38.7% in patients with shock) and varied markedly by indication and use of revascularization procedures. Major IABP complications occurred in only 2.7% of patients, despite median use for three days, and early IABP discontinuation was required in only 2.1% of patients. CONCLUSIONS: With contemporary advances in device technology, insertion technique, and operator experience, IABP counterpulsation may be successfully employed for a wide variety of conditions in the AMI setting, providing significant hemodynamic support with rare major complications in a high-risk patient population.